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Abstract 

We establish rigidity (or uniqueness) theorems for nc automor¬ 
phisms which are natural extensions of clasical results of H. Cartan 
and are improvements of recent results. We apply our results to nc- 
domains consisting of unit balls of rectangular matrices. 


1 Introduction 

Holomorphic automorphisms of domains in C'^ have been studied since the 
work of H. and E. Cartan in the 1930’s 0 . 0 . A holomorphic function can 
be thought of as a generalized polynomial, and they can be evaluated not just 
on tuples of complex numbers, but also on tuples of commuting matrices or 
commuting operators whose spectrum is in the domain of the function 1201. 
An nc-function (nc stands for non-commutative) is a generalization of a free 
polynomial, (he. a polynomial in non-commuting variables), and it is natural 
to evaluate them on tuples of matrices or operators. 

To describe nc-functions (following [13] for instance), we must Erst es¬ 
tablish some notation. Let denote the n-by-n complex matrices, and 
the d-tuples of n-by-n matrices. We shall let denote the disjoint union 
Given x = {x^,... ,x^) in and y = {y^,... ,y^) in M^, hj x®y 
we mean the element {x^(By ^,..., x‘^(By'^) of If a: G and s, t G M„, 

by sxt we mean {sx^t,..., sxH). 

*Partially supported by National Science Foundation Grant DMS 1300280 
^Supported by the Science Foundation Ireland under grant 11/RFP/MTH3187. 


1 



A free polynomial pin d variables can be thonglit of as a function defined 
on Mil'll, and as such it has the following properties: 

(i) If X is in M^, then p{x) G M„. 

(ii) If X and y are in then p{x (B y) = p{x) © p{y). 

(hi) If X G and s G M„, is invertible, then p{s = s ^p{x)s. 

Definition 1. An nc-set is a set C such that := fl is an 
open set for each n, and such that is closed with respect to direct sums 
and joint unitary equivalence (he. for all x G fin and for all u unitary in Mn, 
we have u~^xu G fi). If an nc-set fi has the property that fin is connected 
for every n, we shall call it an nc-domain. 

An nc-function is a function on an nc-set that mimics the properties (i) 
- (iii) above of free polynomials. 

Definition 2. An nc-function / on an nc-set fi is a function with the fol¬ 
lowing three properties: 

(i) If X is in fin, then /(x) G Mn (we say / is graded if this occurs). 

(ii) If X and y are in fi, then /(x (B y) = /(x) © f{y). 

(iii) If X G fin, s G Mn is invertible, and s ^xs G fi, then /(s ^xs) = 
s~^f{x)s. 

An nc-map $ on an nc-domain fi C Mt'^^ is a d-tuple of nc-functions. 
If $ is an nc-map on fi that is also a bijection onto fi, we call it an nc- 
automorphism. 

Bounded symmetric domains in have been characterized by E. Cartan 
[6], and in the course of the proof automorphisms of such domains were 
described. 

We are interested in the following questions about nc-automorphisms. 

Question 1. (Rigidity) If <h is an nc-automorphism of fi, is it uniquely deter¬ 
mined by its action on fi fl M(( for some hxed nl 

Question 2. (Extendibility) If F : fiflM^ —)■ fiflM^ is a biholomorphic map 
which respects similarities, is there an nc-automorphism $ : fi —)■ fi such 
that = Fl 

Question 3. What groups can arise as the automorphism group of an nc- 
domain? 

A set fi C MI'^I is called nc-bounded if for each n, there exists a constant 
Mn such that 

V z G fi„, ||z|| < Mn- 


2 


In Theorem 0 we show that one always has rigidity on nc-bonnded do¬ 
mains that contain the origin. For snch domains, the possible antomorphism 
groups are therefore no more than certain subgroups of the automorphism 
groups of bounded domains (and we substantially answer questions [H [2] and 
[3] for circular bounded domains that contain the origin). In Example fT5l we 
show that many different domains can have the same automorphism groups. 

In Section 01 we consider the Cartan domain of type I, the set Rpg of 
p-hy-q contractive matrices. The obvious nc-domain containing this, where 
numbers are replaced by n-by-n matrices, we call Rpg. In Theorem [131 we 
show that when p q, all automorphisms of Rpq extend; but when p = q, 
only those automorphisms that do not involve the transpose extend. 

In this paper, we restrict our attention to nc-bounded domains, as the 
unbounded case is much more complicated (see e.g. Example [9]). 

This note continues work of Popescu in [T6l [T71 [T8] and of Helton, Klep, 
McCullough and Slinglend, in [T2] . 

2 Background on nc-functions 

The recent monograph [13] by D. Kaliuzhnyi-Verbovetskyi & V. Vinnikov 
gives an introduction to nc-functions. Unless an additional hypothesis of 
continuity (or boundedness) is added, nc-functions can behave badly. 

Example 3. Let d = 1, and dehne a function / on Jordan blocks by sending 
a Jordan block with 0 eigenvalues to the zero matrix of the same size, and 
a Jordan block with non-zero eigenvalues to the identity matrix of that size. 
Extend / by direct sums to any matrix in Jordan canonical form, and then 
by similarity to any matrix. The function / is then an nc-function which is 
manifestly discontinuous. 

Let a denote the disjoint union topology on Mil'll: a set [7 is in a if and 
only if t/ n is open for every n. (This topology is called the hnitely open 
topology in [T3]j. 

It was proved in [T] that if an nc-function / on an nc-set H is cr locally 
bounded, in the sense that 

Vz gH, 3[/Gcr s.t. z eU and /|onc/ is bounded, 
then f : Q ^ is cr-cr continuous, and in [10], it was shown that this in 
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turn implied that / was holomorphic, in the sense that 

Vn G N, Vz G fin, V/i G M^, 3 Df{z)[h] = lim-[f{z + th) — f{z)]. (2.1) 

t ^0 t 

Putting these results together, we conclude 

Proposition 4. An nc-map f into an nc-bounded domain is automatically 
a-a continuous, and holomorphic in the sense of fl2.1l) . 

3 Rigidity 

A domain is called circular if it is invariant under multiplication by unimod- 
ular scalars. 

The following lemmas are classical and due to H. Cartan. 

Lemma 5 (H. Cartan ([51 Theoreme VII, p. 30])). Let V <L£A be a bounded 
domain, Zq eT> and (j)\ V a biholomorphic automorphism with 4>{zo) = 
zo and (j)'{zo) = In- Then (f is the identity. 

Lemma 6 ([7], [H Theoreme VI]). If V is a bounded circular domain in C'^ 
containing 0, and F\ V V is a biholomorphic automorphism of T> with 
F(0) = 0, then F is the restriction to T> of an invertible linear map. 

Theorem 7. LetLl be an nc-domain that is nc-bounded. Let^ = (<h^, ..., <h'^) 
be an nc-automorphism of fl. Suppose that for some m G N, we have 
0 G n n M;^ and (<h|o„.)(0) = 0. 

(i) If in addition ($|o,„)^(0) is the identity, then 4) is the identity on all of 

n. 

(a) If instead we suppose also that Q is a circular nc-domain, then there 
is an invertible linear map F on VA such that ^{Z) = (F ® idn){Z) 
for Z E Qn (by which we mean that each d-tuple {^{Z))ij formed from 
the {i,j) coordinates of the d-tuple of n x n matrices ^{Z) is given 
by F{Zij) where Z = (Z^,..., Z^), and Zij = (ZF,..., Zfj) again 
denotes the {i,j) coordinates). 

Proof. (i) By Lemma O is the identity. 

As Qrn = fl n is open, there is some e > 0 such that ii z = 
[z^,. .. ,z'^) G has each \\zm < e, then z G Llm- 
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Now, fix a positive integer k. For G we write for the d- 

tuple G that has Z* = 0 when i ^ j, and whose 

entry is ZF If Z^ is the direct sum of k matrices from M^, and if 
II^■^11 < then $(Z'^) = Zh by the direct sum property of nc-maps. 
As this applies to for |C| < 1 (C £ C), it follows that the directional 
derivative (($|f7j,„,)^(0))(Z'^) = Zh By linearity of the Frechet derivative 
($|f7j.^)'(0) we may drop the restriction that HZ-^H < e. In particular 
the conclusion holds when Z^ G is a diagonal matrix. 

By the chain rule and similarity invariance 

(<!>k„)(s-‘Zs) = s-'('I>ln.„)(^)s 

of the map *h|oj,^ (valid for all sufficiently small Z once s is hxed), we 
must have that (‘h|oj.^)'(0) has the invariance property 

= ^-^((($|o,J'(0))(Z))5 (3.1) 

Choosing Z^ diagonalisable and s such that s~^Z^s is diagonal fl3.ip 
yields s-H(($|n,^)'(0))(Z^'))s = s-^Z^s and hence (($|n,^)'(0))(Z^) = 
Z^ provided Z^ is diagonalisable. By density of the diagonalisable ma¬ 
trices we can then make the same conclusion with Z^ arbitrary and 
then linearity of (d’|oj,^)X0) forces it to be the identity. 

By Lemma [5] again, *h|ofc^ must be the identity. 

Now choose some n, not necessarily a multiple of m, such that is 
non-empty. Let Z G fin- The direct sum of m copies of Z is in flmn, 
and <F(©™^Z) = ©™iZ by the hrst part of the proof. As <h preserves 
direct sums, this means that *h(Z) = Z. 

(ii) By Lemma ini we know ‘h|r 2 „i is linear. If m = 1 we take F to be <I>|q^. 
For m > 1 we need a brief argument to hnd F. 

Similarity invariance 

($|n„.)(s-'Z5) = s-'($|n.)(Z)s 

(guaranteed by Dehnition [2] to hold for Z, s“^Zs G flm) must hold 
globally for Z G in view of linearity. Let F* ^ denote the standard 

matrix units in and choose Z = ..., z'^Ei^i) (so that Z is 

supported on the (1,1) entries). Consider a block diagonal s = 1 © t 
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with 1 in the (1,1) entry but arbitrary invertible {m — 1) x (m — 1) 
block t. Since the matrices that commute with all such 1 © t are those 
of the form F(BaIm-i (for scalars F and a) we see that $|q^(Z) must 
take the form 

{F^{z) © F'^{z) © a'^{z)I„i-i), with z = {z^,..., /), 

for some scalar-valued linear F^,, F'^, a^,... —)■ C. 

However, notice that (by Lemma |6]) we also know that $102^ is linear, 
and by the direct sum property <h(Z © 0) = *h(2') © 0. The similarity 
argument applied to forces = 0 (1 < j < d). Thus 

..., = {F\z)E,,^,..., F\z)E,,,) 

(with z = {z^,..., z‘^)). Using similarity with s a transposition allows 
us to conclude that the same must hold for E^^k replacing Ui i. Taking 
s = In — Ei^k ioT i ^ k we have s~^EiiS = Ea — Eik, and together with 
linearity we deduce the relation with Ek^k replaced by Ei^k- Clearly 
F = (F^,..., —)■ must be invertible (since $ is) and we 

have the desired conclusion on 

By Lemma ini we know is linear for each k eN. For G if 

is diagonal we must have <F(Z-^) of the required form. The similarity 
property and density of the diagonalisable matrices in allows us 
to extend to arbitrary nonzero ZK Then by linearity this extends to 
arbitrary d-tuples Z = {Z^,..., Z’^) e 

Finally if is nonempty for some n we can apply the result just 
obtained for klnm together with the direct sum property for Z©0©- • -©0 
(where Z G M„ and we have (m — 1) zero summands) to establish the 
desired conclusion for <F|n^. □ 

Popescu’s Cartan uniqueness results [T71 §1] can be viewed as similar in 
spirit for the case of special domains (row contractions) to Theorem [71 In the 
case m = 1, the result of [TOl Corollary 4.1 (2)] is part (i) while [TTl Theorem 
21] implies (ii). 

Example 8. The matrix polydisk. This is the set 

D = {x G : ||x-^|| < 1,1 < i < d}. 
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The set of automorphisms of D is the set 

{<h(x) = cr o , m‘^{x‘^)) : a e S^, E Aut(ro)}. (3.2) 

Here is the symmetric group on d variables, and Aut(D) is the Mobius 
group of automorphisms of the disk. Each Mobius transformation of the form 

, , ,n z — a 

m(z) = -— 

1 — az 

extends to matrices in the obvious way: 

m{Z) = - al){l - dZ)-\ 

Indeed, by von Neumann’s inequality, every $ in fl3.2p extends to an auto¬ 
morphism of D. That this comprises everything follows from observing that 
all automorphisms of the polydisk = D fl Mf are of this form, and so 
by Theorem [7] they have a unique extension to higher levels. As they are 
invertible, they must be automorphisms. < 

Example 9. Theorem [7] fails if boundedness is dropped. Consider, for exam¬ 
ple, the nc-set 

n = {(x, ?/, z) G : ||xy — |/x|| < 1}. 


Let 


^{x,y,z) = {x,y,z + h{xy-yx)), 


where h : C —?■ C is any non-constant entire function mapping 0 to 0. Then 
$ is an automorphism, and ‘h|nnMf is the identity, but <h is not the identity 
on level 2. < 


4 Extendibility in 

Let Rpq denote the p-hy-q matrices of norm less than 1. We shall extend this 
to an nc-domain in Mil'll, where d = pq, by 


n=l 


\x 


x^ 

. x«\ 


. x^'^ 


. xP-'y 


<!}. 


7 










Let 7 : —)■ be the map that takes d matrices and makes them into 

a block p-hy-q matrix, hlling in left to right and then top to bottom. Then 
R-pg = {x : ||7(a;)|| < !}• When we speak of an nc automorphism of Rpg, 
strictly speaking we mean an nc automorphism of 7 “^(Rpq). 

In the special case q = 1, Rpi is just the unit ball in C^, and its auto¬ 
morphisms are well-known (see e.g. [ini Thm. 2.2.5]). The set Rig, the 
row-contractions, was studied by G. Popescu in [T51 [IS] and im. 

The automorphisms of Rpg are given by a similar formula to the case of 
the ball. L. Harris showed [9] that they are of the following form. 

Theorem 10. [Harris] Every holomorphic automorphism of Rpg is of the 
form LHA where L is a linear isometric automorphism of Rpg, A is an ele¬ 
ment of Rpg, and 

Ha{x) = {Ip-AA*)-^/\x + A){Ig + A*x)-\lg-A*AY/\ 

First, let us consider that automorphisms that map 0 to 0, which are the 
linear ones. K. Morita [HI classihed the linear isometries of Rpg, and the 
square case differs from the rectangular case, because the transpose is an 
isometry. 

Theorem 11. [Morita] If p ^ q, all linear automorphisms of Rpg are of the 
form X UxV, where U is a p-by-p unitary and V is a q-by-q unitary. If 
p = q, the set of linear automorphisms consists of x UxV and x e-)> UxM. 

The map x UxV extends to the nc automorphism of Rpg given by 
Z I-7 (id (8) U)Z{id ® V)] but the transpose does not extend. 

Lemma 12. If p > 1, the map x ^ x^ does not extend to an nc automor¬ 
phism of Rpp. 

Proof. Using Theorem (3 if the transpose did extend, the extension would 
map (Xij) G Mp(M„(C)) to and so this reduces to the well-known 

fact that the transpose map is not a complete isometry of Mp. □ 

Let us turn now to Ha- The map Ha extends to an nc map from Rpg to 
Rpg given by 

Ha{z) = {in,p-mAA*)-^/\z+mA){in,g+{mA*)z)-\in,g-mA*AY/^ 

(4.1) 

Here, id means idc^^, and denotes idc" 0 C'-- 


A calculation shows that 


id-HA{WyHA{Z) = + 

{In,, - W*Z){In,, + (id 0 A*)Z)-\In,q - id 0 A*A^/^ (4.2) 

Letting W = Z proves that Ha maps Up, to Rpg, and as H^a is the inverse 
of Ha, it must be an automorphism. 

Putting these results together, we get the following theorem. The case 
p = 1 was proved by Popescu HU Thm. 1.5]. The general case was proved 
by Helton, Klep, McCullough and Slinglend [121 Thm. 1.7], though their 
hypotheses are stronger. The linear case was proved by D. Blecher and D. 
Hay [5]. 

Theorem 13. If p ^ q, then every automorphism of Rp, extends uniquely 
to an automorphism of Up,. If p = q, the automorphisms of the form x 
UHa{x)V extend uniquely to Rpp, and the automorphisms of the form x 
UHa{xYV (when p > 1) do not extend to automorphisms o/Rpp. 

By a result of J. Ball and V. Bolotnikov [1], (see also [3] and H) Ha 
extends to an endomorphism of the commuting elements of 7 “^(Rpg) if and 
only if there is some function F so that 

I - Ha{wYHa{z) = F{w)*{I -w*z)F{z) 

as a kernel on Rpg. This is true, as fl4.2p shows. So the nc automorphisms 
of {x G 7 “^(Rpg) : a:\Vl < i,j < d} are the same as the nc 

automorphisms of 7 “^(Rpg). This phenomenon has also been explored in [1]. 

Question 4. The automorphisms of Rpg are not transitive at any level n >2. 
What can one say about the orbits? 

Theorem [13] can be extended slightly. For S a subset of N that is closed 
under addition, let Rpg(S') be dehned by Rpg(S') fl is Rpg fl if n G S', 
and empty otherwise. 

Proposition 14. Let S be any non-empty sub-semigroup of N. Then the 
automorphisms of Rpg (S') are the same as the automorphisms of Rpg, and 
are uniquely determined by their action on any non-empty level. 

Example 15. Extendibility can fail if the pieces of 12 at different levels are not 
somehow alike. For example, let d = 1, and R> 1. Dehne 12 by 12nMi = ©, 
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and = {x : ||a:|| < R}. The automorphisms of f2nMi are the Mobius 

maps, but only multiplication by extends to be an automorphism of 
But there are many other choices of nc-domain D Rn that have the 
same automorphism group. For example, let ri = 1, and let {rn)'^=i be any 
non-decreasing sequence. Dehne by 

fin = (x G M„ : 3s G M„, with ||s||||s“^|| < r„, and ||s“^a:s|| < 1}. (4.3) 

Then is an nc-domain, and its automorphism group is the set of Mobius 
maps. <] 

Question 5. Let 17 C be a bounded symmetric domain. Is there an nc- 
domain C Mil'll such that fli = U and such that the automorphism group 
of 17 equals the automorphism group of U1 
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